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Abstract 

ISX  1.1  was  conducted  in  the  Summer  1998  in  concert  with  the  USAF  Expeditionary  Force  Ex¬ 
periment  (EFX)  98.  The  broad  objectives  of  ISX  1.1  were  two-fold.  Substantively,  ISX  1.1  was 
to  confirm  the  hypothesis  that  information  from  multiple  sources  could  be  fused  into  engagement 
quality  data  and  distributed  in  real  time  to  achieve  joint  suppression  of  enemy  air  defenses 
(JSEAD)  mission  objectives  against  mobile  medium-to-high  altitude  surface  to  air  missiles 
(SAMs).  Second,  since  this  was  the  first  in  a  planned  set  of  experiments,  there  was  interest  in  de¬ 
riving  lessons  learned  to  support  the  efficient  and  effective  performance  of  future  ISXs. 

Based  on  an  assessment  of  ISX  1.1,  several  significant  experimental  successes  and  residual  chal¬ 
lenges  have  been  identified.  These  successes  include  the  innovative  use  of  models,  the  implemen¬ 
tation  of  a  creative  experimental  design,  and  the  data  collection  efforts.  The  major  residual  chal¬ 
lenges  include  the  difficulties  in  “piggybacking”  on  other  experimental  activities  and  the  impor¬ 
tance  in  viewing  an  experiment  in  the  context  of  a  campaign  in  which  a  variety  of  assessment 
techniques  are  employed  (e.g.,  model,  test,  model,...). 

1.  Background 

In  response  to  Joint  Vision  2010  (promulgated  by  the  Chairman,  Joint  Chiefs  of  Staff)  [Joint 
Chiefs,  1996],  a  plan  was  generated  by  the  Director,  J6,  Joint  Staff,  to  conduct  a  family  of  Infor¬ 
mation  Superiority  Experiments  (ISXs).  These  experiments  were  structured  to  address  many  of 
the  highest  priority  issues  identified  during  the  Quadrennial  Defense  Review  [Cohen,  1997].  To 
initiate  this  experimentation  program,  ISX  1.1  was  conducted  late  in  the  Summer  of  1998  in  con¬ 
cert  with  the  USAF  Expeditionary  Force  Experiment  (EFX)  98.  The  broad  objectives  of  ISX  1.1 
were  two-fold.  Substantively,  ISX  1.1  was  to  confirm  the  hypothesis  that  information  from  multi¬ 
ple  sources  could  be  fused  into  engagement  quality  data  and  distributed  in  real  time  to  achieve 
joint  suppression  of  enemy  air  defenses  (JSEAD)  mission  objectives  against  mobile  medium-to- 
high  altitude  surface  to  air  missiles  (SAMs).  Second,  since  this  was  the  first  in  a  planned  set  of 
experiments,  there  was  interest  in  deriving  lessons  learned  to  support  the  efficient  and  effective 
performance  of  future  ISXs. 
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In  support  of  the  second  of  these  objectives,  the  Information  and  C2  Systems  Technical  Commit¬ 
tee  (ICS  TC)  of  the  American  Institute  for  Aeronautics  and  Astronautics  (AIAA)1  volunteered  to 
observe  ISX  1.1  and  to  derive  lesson  learned  to  enhance  future  ISXs.  The  focus  of  the  ICS  TC 
activities  was  on  the  overall  experimental  process  that  was  employed  from  a  lifecycle  perspective, 
vice  the  substantive  assessment  of  the  experiment’s  primary  technical  hypothesis.  To  realize  that 
objective,  representatives  from  the  group  were  briefed  on  the  planning  for  ISX  1.1  and  EFX  98, 
observed  the  activities  from  multiple  vantage  points,  and  reviewed  relevant  after  action  reports. 
Based  on  these  activities,  several  significant  experimental  successes  and  residual  challenges  have 
been  identified. 

2.  Primary  Findings 

Three  major  successes  were  achieved  in  ISX  1.1.  First,  the  ISX  1.1  planning  team  used  models 
creatively  to  help  design  and  plan  the  live  experiment.  By  doing  so,  the  team  was  able  to  provide 
insights  into  important  JSEAD  issues,  suggest  hypotheses,  reduce  the  scope  of  the  problem,  iden¬ 
tify  critical  parameters,  suggest  data  to  be  collected,  identify  Measures  of  Merit  to  compute,  and 
identify  selected  risks.  Second,  they  designed  the  live  experiments  as  a  sequence  of  nearly  inde¬ 
pendent  opportunities.  These  mini-tests  provided  useful  data  even  though  few  replications  of  the 
entire  operational  “threads”  were  achieved.  Third,  they  learned  valuable  lessons  about  data  col¬ 
lection.  These  included  the  appropriate  composition  of  the  data  collection  team,  the  kind  of  data 
to  collect,  and  the  appropriate  methods  to  collect  data. 

The  experience  also  underscored  several  challenges  that  remain  to  be  addressed.  First,  the  ISX  1.1 
hypothesis  was  not  tested  adequately.  This  was  due,  in  large  measure,  to  shortfalls  in  backup 
planning  (e.g.,  inability  to  cope  with  adverse  weather,  failures  in  technology  initiatives)  and  insuf¬ 
ficient  training  of  participants.  Second,  the  concept  of  “piggybacking”  on  EFX  98  proved  ineffec¬ 
tive.  There  are  several  reasons  for  this  shortfall  which  will  be  touched  on  in  the  “Recommenda¬ 
tions”  section  of  this  paper.  It  should  be  noted,  however,  that  ISX  1.1  did  not  interfere  with  EFX 
and  ISX  1.1  participation  served  to  enhance  EFX  98  (e.g.,  it  increased  EFX  live  fly,  significantly; 
it  added  additional  live  fly  platforms;  it  shared  data  collection,  data,  and  lessons  learned  with 
EFX). 

Many  of  the  problems  identified  here  have  been  observed  and  recorded  in  prior  experiments. 
Since  they  have  not  been  corrected,  however,  they  are  denoted  as  “lessons  recorded”.  Hopefully, 
they  will  become  “lessons  learned”  and  acted  upon  proactively  in  future  experiments. 

3.  Selected  Lessons  Recorded 

The  assessment  of  ISX  1.1  revealed  that  there  are  a  host  of  potential  problems  that  can  adversely 
affect  a  joint  experiment  in  the  areas  of  planning,  preparation,  execution,  data  collection,  and  ad¬ 
ministrative  actions.  Although  no  single  problem  is  generally  “lethal”  to  an  experiment,  they  can 


1  The  ICS  TC  of  the  AIAA  is  a  group  of  approximately  thirty  senior  representatives  from  the  C2  community  drawn  from  the 
government,  industry.  Federally  Funded  Research  and  Development  Centers  (FFRDCs),  and  academia.  It  focuses  on  the  is¬ 
sues  associated  with  creating,  evaluating,  and  evolving  systems  of  systems.  Consistent  with  that  focus,  the  group  is  currently 
developing  a  Code  of  Best  Practice  (COBP)  for  joint  C2  experimentation  for  the  Office  of  the  Secretary  of  Defense. 


be  like  duck  bites.  If  you  experience  enough  of  them,  the  cumulative  effect  can  be  devastating. 
The  following  discussion  identifies  these  “duck  bites”  in  five  key  areas:  planning,  preparation, 
execution,  data  collection,  and  administrative  actions. 

3.1  Planning 

A  total  of  seven  planning  issues  were  recorded  based  on  observations  of  ISX  1.1. 

•  Limit  the  Number  of  Initiatives  Introduced.  Many  members  of  the  community  regard  events  like 
EFX  as  an  important  showcase  for  their  initiatives.  As  a  consequence,  in  excess  of  50  initiatives 
were  subsumed  within  EFX,  many  of  which  were  added  very  late  in  the  planning  process.  There 
are  several  significant  consequences  of  this  trend.  First,  it  substantially  increases  the  amount  of 
training  needed  by  all  participants  in  the  experiment,  and  the  time  and  resources  needed  to  ac¬ 
complish  the  training.  Second,  by  introducing  many  unproved  concepts,  systems,  and  technology, 
it  provides  more  opportunities  for  failure.  This  is  particularly  significant  when  the  initiatives  are  on 
the  experiment’s  critical  path.  Finally,  there  is  a  significant  scientific  downside.  Each  initiative  in¬ 
creases  experimental  variability  and  decreases  the  amount  of  control  that  can  be  imposed  on  the 
experiment.  Consequently,  it  is  recommended  that  the  number  of  initiatives  incorporated  in  an  ex¬ 
periment  be  curtailed  to  a  manageable  level.  In  addition,  a  “Good  Idea  Cut-Off  (GICO)”  date 
should  be  established  at  least  six  months  prior  to  the  start  of  the  experimental  event  and  enforced 
rigorously. 

•  Maintain  Tight  Control  of  the  Master  Scenario  Event  List  (MSEL).  In  ISX  1.1  players  at  key 
experimentation  nodes  had  ready  access  to  the  MSEF  (e.g.,  frequently,  Air  Operations  Center 
personnel  had  knowledge  about  the  schedule  of  attack  aircraft  flights,  affecting  their  decisions). 
Closer  hold  should  be  maintained  over  this  information  to  avoid  confounding  the  results  of  the 
experiment. 

•  Institute  Enhanced  Configuration  Control.  Changes  to  key  EFX  systems  were  made  as  late  as  a 
few  days  prior  to  the  start  of  the  live  experiment.  For  example,  a  new  version  of  the  Theater  Bat¬ 
tle  Management  Core  System  (TBMCS)  was  installed  the  week  before  the  start  of  live  fly.  Such 
actions  make  it  exceedingly  difficult  to  implement  and  maintain  configuration  control  on  all  the 
hardware  and  software  that  are  employed. 

•  Select  an  Adequate  Test  Range.  It  is  critical  that  a  venue  for  the  experiment  be  selected  that 
provides  adequate  air  and  ground  space.  The  Eglin  AFB  Range  posed  a  number  of  problems  that 
adversely  affected  ISX  1.1.  First,  the  environment  featured  very  flat,  benign  terrain  and  road  net¬ 
works  that  are  not  representative  of  likely  operational  environments.  Second,  restrictions  were 
placed  on  threat  radar  emissions  that  limited  the  realism  of  their  use  in  the  experiment.  Third,  un¬ 
manned  air  vehicles  (UAVs)  were  required  to  fly  above  4Kft,  far  away  from  target  vehicles.  This 
made  it  nearly  impossible  to  use  these  assets  realistically,  particularly  given  the  heavy  cloud  cover 
that  was  experienced  during  the  experiment.  Finally,  the  geometry  of  the  range  was  not  represen¬ 
tative  of  the  sensor  -  threat  ranges  that  would  be  anticipated  operationally. 


•  Schedule  ISR  Assets  with  Sufficient  Slack  Time.  Experimental  artificialities  called  for  selected 
critical  radar  emissions  to  occur  at  precise,  specified  times.  However,  on  several  occasions,  the 
intelligence,  surveillance,  and  reconnaissance  (ISR)  assets  were  not  on  station  when  those  trig¬ 
gering  events  occurred.  This  demonstrates  that,  unless  sufficient  slack  time  is  planned  for,  small 
scheduling  mishaps  can  have  exceedingly  deleterious  effects  on  experiments. 

•  Schedule  Offensive  Information  Operations  Judiciously.  In  a  realistic  environment,  US  C4ISR 
assets  would  likely  be  subject  to  Information  Warfare  attacks.  However,  the  fragility  of  the  ex¬ 
perimental  C4ISR  system  is  such,  that  it  is  of  dubious  value  to  attack  it  during  an  experiment.  If 
such  attacks  are  deemed  necessary,  they  should  be  scheduled  to  occur  near  the  end  of  a  game  pe¬ 
riod  to  allow  recovery  during  breaks. 

•  Plan  for  Work  Arounds.  During  the  course  of  the  experiment,  numerous  problems  were  caused 
by  adverse  weather  and  equipment  failures.  These  problems  were  compounded  by  the  failure  to 
anticipate  such  events  and  to  develop  back-up  plans  to  mitigate  their  effects. 

3.2  Preparation 

A  total  of  four  planning  issues  were  identified  based  on  observations  of  ISX  1.1. 

•  Indoctrinate  Participants  on  Experiment  Objectives.  It  is  important  to  indoctrinate  the  players 
about  the  objectives  of  the  experiment.  In  the  absence  of  such  an  understanding,  the  participants 
are  prone  to  revert  back  to  the  objective  of  “winning  the  war”  or  to  consider  the  event  to  be  a 
training  exercise.  Under  those  circumstances,  they  do  not  manifest  the  behavior  that  is  appropriate 
to  the  evaluation  of  new  operational  concepts. 

•  Schedule  Adequate  Training  Time.  It  is  apparent  that  all  of  the  participants  in  the  exercise  (to 
include,  inter  cdia,  the  exercise  director,  the  players,  and  data  collectors)  must  be  fully  conversant 
about  their  roles,  equipment,  and  processes  before  the  experiment  begins.  However,  this  critical 
need  is  frequently  the  first  casualty  of  experimentation  as  the  press  of  time  continually  reduces 
the  period  that  is  allocated  to  this  function.  In  addition,  it  is  naive  to  merely  allocate  a  fixed  period 
of  time  for  training.  It  is  important  to  establish  proficiency  criteria  that  must  be  satisfied  before  the 
experiment  begins.  This  was  particularly  important  in  EFX  98  where  key  individuals  were  deliber¬ 
ately  assembled  from  across  the  USAF  to  foster  innovative  ideas.  Thus,  adequate  training  was 
vital  to  ensure  that  they  could  work  together  as  an  effective  team  prior  to  the  beginning  of  the  ex¬ 
periment. 

•  Check  Out  Communications  Prior  to  Live  Fly.  At  the  outset  of  ISX  1.1,  it  soon  became  appar¬ 
ent  that  there  was  a  lack  of  connectivity  between  several  key  nodes  (e.g.,  between  the  Time  Criti¬ 
cal  Target  (TCT)  Cell  and  the  UAV  ground  site).  A  careful  check  out  of  the  communications 
prior  to  the  live  fly  would  have  served  to  identify  those  issues. 

•  Establish  and  Maintain  Time  Synchronization.  During  ISX  1.1  it  was  noted  that  the  “official 
time”  differed  at  key  command  and  control  nodes  by  +/-  one  minute.  Since  timelines  were  a  criti¬ 
cal  element  of  the  experiment,  these  discrepancies  posed  a  significant  problem  in  the  analysis  pro- 


cess.  In  the  future,  a  single  source  of  time  (e.g.,  the  Global  Positioning  System)  should  be  em¬ 
ployed  by  all  participants. 


3.3  Execution 

A  total  of  three  execution  issues  were  identified  based  on  observations  of  ISX  1.1. 

•  Synchronize  Live  and  Simulated  Events.  During  the  first  week  of  the  experiment,  the  partici¬ 
pants  were  exposed  to  an  air  picture  that  juxtaposed  live  flights  (over  the  Eglin  AFB  Range)  and 
simulated  flights  (in  a  hypothetical  Southwest  Asia  scenario).  The  logical  inconsistency  between 
the  live  and  simulated  events  posed  significant  problems  for  the  participants. 

•  Maintain  an  Equipment  Status  Board.  During  the  course  of  ISX  1.1,  there  was  frequently  con¬ 
fusion  about  the  status  of  key  systems,  and  the  underlying  reasons  for  problems.  To  deal  with 
those  issues  a  status  board  is  needed  that  would  identify  those  systems  that  are  experiencing  diffi¬ 
culties  and  the  underlying  reasons  for  those  problems  (e.g.,  adverse  weather,  mechanical  prob¬ 
lems,  maintenance). 

•  Establish  a  Responsive,  Flexible  White  Cell.  There  were  several  instances  where  a  more  re¬ 
sponsive,  flexible  White  Cell  would  have  ameliorated  execution  problems.  For  example,  on  sev¬ 
eral  occasions,  the  experiment  controller  arbitrarily  made  decisions,  causing  MSEF  deviations,  yet 
failed  to  announce  the  changes  to  the  Assessment  Team.  In  addition,  there  were  cases  where  de¬ 
partures  from  the  original  plan  could  have  been  compensated  for  by  manually  inserting  cues  into 
the  experiment.  However,  no  mechanism  was  created  to  implement  such  manual  over-rides. 

3.4.  Data  Collection 

A  total  of  six  data  collection  issues  were  identified  based  on  observations  of  ISX  1.1. 

•  Provide  Sufficient  Data  Collection  Tools.  One  frequently  praised  initiative  during  EFX  98  was 
the  Collaborative  Virtual  Workstation  (CVW).  Participants,  using  this  tool,  were  able  to  meet  in 
virtual  “rooms”  in  which  they  could  plan  collaboratively.  However,  observers  who  did  not  have 
access  to  CVW  were  unable  to  determine  when  important  decisions  were  made  or  what  informa¬ 
tion  those  decisions  were  based  on.  If  these  tools  are  to  be  used  extensively  in  future  operations, 
data  collectors  must  have  sufficient  CVW  terminals. 

•  Install,  Check-out  All  Data  Collection  Equipment  Early.  Among  the  data  collection  assets  em¬ 
ployed  during  ISX  1.1  were  computers,  printers,  recording  devices,  storage  devices,  communica¬ 
tions,  electrical  outlets,  dataforms,  databases,  and  assorted  software  applications.  The  complexity 
of  these  assets  is  such  that  early  installation  and  quality  assurance  is  mandatory.  To  deal  with  in¬ 
evitable  malfunctions,  it  is  critical  that  at  least  one  trained  technician  be  included  with  the  assess¬ 
ment  teams. 

•  Train  Data  Collectors/Assessors.  In  view  of  the  complexity  of  the  experimental  process  and  the 
simulation  tools  involved,  it  is  critical  that  data  collectors  and  assessors  are  provided  with  ade- 


quate  training  prior  to  the  on-set  of  the  experiment.  As  one  facet  of  the  training,  these  individuals 
need  a  better  understanding  of  the  planned  information  flow  and  the  activities  being  performed. 
This  should  provide  them  with  “enhanced  situational  awareness”  so  that  they  are  better  prepared 
to  react  to  unexpected  developments. 

•  Debrief  Air  Crews  Using  Assessors.  In  several  cases,  air  crews  were  subject  to  debriefings  that 
were  appropriate  for  an  actual  operational  event  or  a  training  event.  Thus,  they  did  not  capture 
key  insights  that  were  appropriate  to  an  experiment. 

•  Select  Data  Collectors/Assessors  With  the  Proper  Backgrounds.  In  several  instances,  data  col¬ 
lectors  and  assessors  were  selected  who  lacked  the  proper  background  for  the  tasks  and  who 
were  unfamiliar  with  the  key  systems  in  the  experiment.  Consequently,  they  were  not  sensitive  to 
key  issues  and  were  not  attuned  to  some  of  the  unique  demands  of  the  experiment. 

•  Develop  Rapport  Between  the  Players  and  the  Data  Collectors/Assessors.  As  one  vehicle  for 
establishing  this  rapport,  data  collectors  should  explain  to  the  players  the  experiment’ s  objectives 
and  data  they  plan  to  collect  prior  to  the  collection  of  real  data. 

3.5  Administrative  Actions 

A  total  of  six  administrative  issues  were  identified  based  on  observations  of  ISX  1.1.  Although 
many  of  these  issues  appear  to  be  minor,  they  can  undermine  the  viability  of  the  experiment  if  they 
are  not  anticipated  and  dealt  with  adequately. 

•  Take  Care  in  Scheduling  Live  Fly  Events.  EFX  98  was  scheduled  to  take  place  in  Florida  during 
the  hurricane  season.  Although  the  precise  impact  was  unpredictable,  it  is  not  surprising  that  ad¬ 
verse  weather  severely  degraded  the  quality  of  the  experiment.  As  examples,  Hurricane  Earl  trun¬ 
cated  Spiral  Three,  curtailing  training  on  new  initiatives  and  concepts  of  operations;  a  subsequent 
tropical  depression  grounded  aircraft  on  one  afternoon  of  the  experiment;  and  heavy  cloud  cover 
and  rain  prevented  the  UAV  from  imaging  the  tasked  targets  during  several  other  runs. 

•  Schedule  Critical  Targeting  Events  Judiciously.  Several  time  critical  events  (e.g.,  activities  as¬ 
sociated  with  simulated  SAMs  and  SCUDs)  were  scheduled  to  occur  at  1100  to  1230  local  time. 
Unfortunately,  many  of  the  people  in  the  TCT  cell  were  at  lunch  during  that  period,  reducing  the 
cell’s  ability  to  respond  to  the  events. 

•  Provide  Assistance  on  Classification  Issues.  During  the  course  of  the  experiment  ,  issues  fre¬ 
quently  emerged  on  what  was  classified  and  what  was  not.  To  deal  with  such  issues,  an  authority 
on  classification  issues  needs  to  be  designated  and  made  available. 

•  Enforce  Discipline  in  Issuing  Badges.  In  ISX  1.1,  a  restricted  set  of  badge  types  was  issued. 
These  included  distinctive  badges  for  controllers,  assessors,  and  participants/players.  Unfortu¬ 
nately,  many  contractors  and  visitors  were  provided  with  participant/player  badges.  Thus,  it  made 
it  difficult  for  data  collectors  to  determine  who/what  they  should  be  observing. 


•  Impose  Greater  Control  on  Visitors.  Visitors  frequently  interfered  with  the  play  of  the  experi¬ 
ment  and  the  collection  of  data.  Even  though  it  is  difficult  to  impose  control  on  such  conduct  if 
the  visitors  are  quite  senior,  it  is  mandatory  if  a  meaningful  experiment  is  to  be  conducted. 

•  Publish  a  Directory  of  Participants.  It  was  frequently  quite  difficult  to  communicate  to  other 
participants  during  this  geographically  distributed  experiment.  To  ameliorate  this  problem,  a  di¬ 
rectory  is  needed  for  key  participants  that  provides  key  voice  and  FAX  phone  numbers  (e.g., 
commercial,  DSN)  and  e-mail  addresses. 

4.  Primary  Recommendations 

Three  primary  recommendations  were  developed  based  on  the  ICS  TC  assessment.  First,  it  is  rec¬ 
ognized  that  resource  limitations  will  inevitably  prompt  future  experimenters  to  “piggyback”  on 
planned  experiments  or  exercises.  However,  it  is  recommended  that  a  piggyback  approach  should 
be  attempted  only  if  there  is  high  level  visibility  and  leverage;  formal  agreement  to  collaborate; 
sufficient  influence  on  the  scenario  to  tailor  it;  early  involvement  in  the  planning  process;  sufficient 
resources  for  training,  data  collection,  and  analysis;  and  a  robust  experimentation  environment, 
including  an  adequate  number  and  variety  of  experimental  events,  instrumentation,  and  free  play. 
It  must  be  emphasized  that  “piggybacking”  is  a  viable  concept  only  if  it  is  perceived  to  be  in  the 
mutual  self-interest  of  all  parties  (i.e.,  a  “win-win”  strategy). 

Second,  the  experimental  effort  should  be  viewed  as  a  campaign  in  which  a  variety  of  assessment 
techniques  are  employed  and  orchestrated  (e.g.,  model,  analyze,  test,  model,...).  Consistent  with 
this  concept,  adequate  resources  should  be  provided  for  the  pre-  and  post-live  test  activities.  In 
addition,  the  logical  relationships  among  individual  experimental  events  must  be  understood  and 
documented  through  a  campaign  plan  [Note:  this  concept  is  discussed  at  greater  length.  [MORS, 
1999], 

Finally,  there  is  a  need  for  a  Code  of  Best  Practices  (COBP)  to  guide  future  joint  C2  experimen¬ 
tation  efforts.  The  ICS  TC  is  in  the  process  of  developing  such  a  COBP  and  it  should  be  com¬ 
pleted  during  the  Summer  1999. 
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